2026-09-21 01:03:35,785 [ 182810 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2026-09-21 01:03:35,785 [ 182810 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2026-09-21 01:03:35,785 [ 182810 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2026-09-21 01:03:35,785 [ 182810 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_39dq2n --privileged --dns-search='.' --memory=30703239168 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel0_1.jsonl --report-log-exclude-logs-on-passed-tests test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [5 items] scheduling tests via LoadFileScheduling test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_database_delta/test.py::test_complex_table_schema test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active [gw3] [ 20%] SKIPPED test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active [gw0] [ 40%] ERROR test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables [gw0] [ 60%] ERROR test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables [gw0] [ 80%] ERROR test_database_delta/test.py::test_multiple_schemes_tables [gw1] [100%] ERROR test_backup_restore_on_cluster/test_different_versions.py::test_different_versions ==================================== ERRORS ==================================== _________________ ERROR at setup of test_complex_table_schema __________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 3508 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ~~~~~~~~~~~~~~~~~~~~~ Stack of (140543218107968) ~~~~~~~~~~~~~~~~~~~~~ File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 411, in _perform_spawn reply.run() File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 341, in run self._result = func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 1160, in _thread_receiver msg = Message.from_io(io) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 567, in from_io header = io.read(9) # type 1, channel 4, payload 4 File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 534, in read data = self._read(numbytes - len(buf)) ------------------------------ Captured log setup ------------------------------ 2026-09-21 01:03:42.373000 [ 590 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.405000 [ 590 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.405000 [ 590 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-09-21 01:03:42.405000 [ 590 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-09-21 01:03:42.405000 [ 590 ] DEBUG : Command:[docker network prune --force] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.435000 [ 590 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV DOCKER_KERBEROS_KDC_TAG 9391ecdee8d7 (cluster.py:424, __init__) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV CLICKHOUSE_TESTS_SERVER_BIN_PATH /clickhouse (cluster.py:424, __init__) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV MSAN_OPTIONS abort_on_error=1 poison_in_dtor=1 (cluster.py:424, __init__) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV JAVA_TOOL_OPTIONS -Djdk.attach.allowAttachSelf=true (cluster.py:424, __init__) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV TSAN_OPTIONS halt_on_error=1 abort_on_error=1 history_size=7 memory_limit_mb=46080 second_deadlock_stack=1 (cluster.py:424, __init__) 2026-09-21 01:03:42.438000 [ 590 ] DEBUG : ENV HOSTNAME b7b71190c1db (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV SHLVL 0 (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV HOME /root (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV OLDPWD / (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV DOCKER_HELPER_TAG 5dc43a6382f0 (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV PYTHONUNBUFFERED 1 (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV DOCKER_PYTHON_BOTTLE_TAG d862517635bf (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV UBSAN_OPTIONS print_stacktrace=1 (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV PYTEST_ADDOPTS --dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel0_1.jsonl --report-log-exclude-logs-on-passed-tests test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables -vvv (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV COMPOSE_HTTP_TIMEOUT 600 (cluster.py:424, __init__) 2026-09-21 01:03:42.439000 [ 590 ] DEBUG : ENV DOCKER_MYSQL_PHP_CLIENT_TAG 88be89c1e3b6 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_DOTNET_CLIENT_TAG 11de0b29a15d (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV CLICKHOUSE_TESTS_CLIENT_BIN_PATH /clickhouse (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_MYSQL_JS_CLIENT_TAG 41ba7c2ec2a1 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV PATH /spark-3.3.2-bin-hadoop3/bin:/opt/gdb/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_KERBERIZED_HADOOP_TAG latest (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_CHANNEL stable (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_CLIENT_TIMEOUT 300 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_POSTGRESQL_JAVA_CLIENT_TAG a4eff5c7f4d6 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_NGINX_DAV_TAG b55ac9cd7519 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_MYSQL_GOLANG_CLIENT_TAG 9bec2a638e6e (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV PWD /ClickHouse/tests/integration (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV DOCKER_MYSQL_JAVA_CLIENT_TAG 766bff31cfe4 (cluster.py:424, __init__) 2026-09-21 01:03:42.440000 [ 590 ] DEBUG : ENV CLICKHOUSE_TESTS_BASE_CONFIG_DIR /clickhouse-config (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV TZ Etc/UTC (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV JAVA_PATH /usr/lib/jvm/java-11-openjdk-amd64/bin/java (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV DOCKER_BASE_TAG 5ccda723c1fc (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV SPARK_HOME /spark-3.3.2-bin-hadoop3 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV LC_CTYPE C.UTF-8 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV INTEGRATION_TESTS_RUN_ID 1 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV WORKER_FREE_PORTS 30000 30001 30002 30003 30004 30005 30006 30007 30008 30009 30010 30011 30012 30013 30014 30015 30016 30017 30018 30019 30020 30021 30022 30023 30024 30025 30026 30027 30028 30029 30030 30031 30032 30033 30034 30035 30036 30037 30038 30039 30040 30041 30042 30043 30044 30045 30046 30047 30048 30049 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV PYTEST_XDIST_TESTRUNUID 15e08d5cff3e4365b16a587a2e5ac8b5 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV PYTEST_XDIST_WORKER gw0 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV PYTEST_XDIST_WORKER_COUNT 10 (cluster.py:424, __init__) 2026-09-21 01:03:42.441000 [ 590 ] DEBUG : ENV PYTEST_CURRENT_TEST test_database_delta/test.py::test_complex_table_schema (setup) (cluster.py:424, __init__) 2026-09-21 01:03:42.442000 [ 590 ] DEBUG : CLUSTER INIT base_config_dir:/clickhouse-config (cluster.py:724, __init__) 2026-09-21 01:03:42.442000 [ 590 ] DEBUG : clickhouse_start_command: clickhouse server --config-file=/etc/clickhouse-server/{main_config_file} --log-file=/var/log/clickhouse-server/clickhouse-server.log --errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log (cluster.py:1662, add_instance) 2026-09-21 01:03:42.442000 [ 590 ] DEBUG : Cluster name: project_name:roottestdatabasedelta-gw0. Added instance name:node1 tag:latest base_cmd:['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env', '--project-name', 'roottestdatabasedelta-gw0', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml'] docker_compose_yml_dir:/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/ (cluster.py:1948, add_instance) 2026-09-21 01:03:42.443000 [ 590 ] INFO : Starting cluster... (test.py:42, started_cluster) 2026-09-21 01:03:42.443000 [ 590 ] INFO : Running tests in /ClickHouse/tests/integration/test_database_delta/test.py (cluster.py:2738, start) 2026-09-21 01:03:42.443000 [ 590 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-09-21 01:03:42.471000 [ 590 ] DEBUG : Docker networks for project roottestdatabasedelta-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:03:42.502000 [ 590 ] DEBUG : Docker containers for project roottestdatabasedelta-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:03:42.532000 [ 590 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw0 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:03:42.532000 [ 590 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:03:42.560000 [ 590 ] DEBUG : Docker networks for project roottestdatabasedelta-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:03:42.588000 [ 590 ] DEBUG : Docker containers for project roottestdatabasedelta-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:03:42.620000 [ 590 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw0 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:03:42.620000 [ 590 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.648000 [ 590 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:03:42.648000 [ 590 ] DEBUG : No running containers for project: roottestdatabasedelta-gw0 (cluster.py:879, cleanup) 2026-09-21 01:03:42.648000 [ 590 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:03:42.676000 [ 590 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:03:42.676000 [ 590 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.706000 [ 590 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:147, run_and_check) 2026-09-21 01:03:42.706000 [ 590 ] DEBUG : Exitcode:1 (cluster.py:149, run_and_check) 2026-09-21 01:03:42.706000 [ 590 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:03:42.706000 [ 590 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.736000 [ 590 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.737000 [ 590 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-09-21 01:03:42.737000 [ 590 ] DEBUG : Setup directory for instance: node1 (cluster.py:2758, start) 2026-09-21 01:03:42.738000 [ 590 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 01:03:42.738000 [ 590 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 01:03:42.739000 [ 590 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 01:03:42.739000 [ 590 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 01:03:42.740000 [ 590 ] DEBUG : Copy custom test config files [] to /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 01:03:42.740000 [ 590 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/database (cluster.py:4758, create_dir) 2026-09-21 01:03:42.740000 [ 590 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/logs (cluster.py:4769, create_dir) 2026-09-21 01:03:42.740000 [ 590 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-09-21 01:03:42.741000 [ 590 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env (cluster.py:96, _create_env_file) 2026-09-21 01:03:42.741000 [ 590 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 01:03:42.741000 [ 590 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 01:03:42.741000 [ 590 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 01:03:42.742000 [ 590 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 01:03:42.755000 [ 590 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-09-21 01:03:42.756000 [ 590 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env --project-name roottestdatabasedelta-gw0 --file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:08:42.760000 [ 590 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env', '--project-name', 'roottestdatabasedelta-gw0', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-09-21 01:08:44.007000 [ 590 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env --project-name roottestdatabasedelta-gw0 --file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:13:44.013000 [ 590 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env', '--project-name', 'roottestdatabasedelta-gw0', '--file', '/ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-09-21 01:13:45.872000 [ 590 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/.env --project-name roottestdatabasedelta-gw0 --file /ClickHouse/tests/integration/test_database_delta/_instances-1-gw0/node1/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.376000 [ 590 ] DEBUG : Failed to start cluster: (cluster.py:3169, start) 2026-09-21 01:18:42.376000 [ 590 ] DEBUG : Timeout >900.0s (cluster.py:3170, start) 2026-09-21 01:18:42.380000 [ 590 ] DEBUG : Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2779, in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) File "/ClickHouse/tests/integration/helpers/retry_decorator.py", line 20, in inner func(*args, **kwargs) File "/ClickHouse/tests/integration/helpers/cluster.py", line 132, in run_and_check res = subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 505, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib/python3.10/subprocess.py", line 1154, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/lib/python3.10/subprocess.py", line 2021, in _communicate ready = selector.select(timeout) File "/usr/lib/python3.10/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 313, in handler timeout_sigalrm(item, settings) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 498, in timeout_sigalrm pytest.fail("Timeout >%ss" % settings.timeout) File "/usr/local/lib/python3.10/dist-packages/_pytest/outcomes.py", line 198, in fail raise Failed(msg=reason, pytrace=pytrace) Failed: Timeout >900.0s (cluster.py:3171, start) 2026-09-21 01:18:42.381000 [ 590 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-09-21 01:18:42.381000 [ 590 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:18:42.409000 [ 590 ] DEBUG : Docker networks for project roottestdatabasedelta-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:18:42.430000 [ 590 ] DEBUG : Docker containers for project roottestdatabasedelta-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:18:42.454000 [ 590 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw0 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:18:42.455000 [ 590 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.479000 [ 590 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:18:42.479000 [ 590 ] DEBUG : No running containers for project: roottestdatabasedelta-gw0 (cluster.py:879, cleanup) 2026-09-21 01:18:42.479000 [ 590 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:18:42.499000 [ 590 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:18:42.499000 [ 590 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.526000 [ 590 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:147, run_and_check) 2026-09-21 01:18:42.526000 [ 590 ] DEBUG : Exitcode:1 (cluster.py:149, run_and_check) 2026-09-21 01:18:42.526000 [ 590 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:18:42.526000 [ 590 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.558000 [ 590 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:18:42.558000 [ 590 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-09-21 01:18:42.559000 [ 590 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-09-21 01:18:42.559000 [ 590 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:18:42.583000 [ 590 ] DEBUG : Docker networks for project roottestdatabasedelta-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:18:42.614000 [ 590 ] DEBUG : Docker containers for project roottestdatabasedelta-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:18:42.647000 [ 590 ] DEBUG : Docker volumes for project roottestdatabasedelta-gw0 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:18:42.647000 [ 590 ] DEBUG : Command:[docker container list --all --filter name='^/roottestdatabasedelta-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.676000 [ 590 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:18:42.677000 [ 590 ] DEBUG : No running containers for project: roottestdatabasedelta-gw0 (cluster.py:879, cleanup) 2026-09-21 01:18:42.677000 [ 590 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:18:42.704000 [ 590 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:18:42.704000 [ 590 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.733000 [ 590 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:147, run_and_check) 2026-09-21 01:18:42.733000 [ 590 ] DEBUG : Exitcode:1 (cluster.py:149, run_and_check) 2026-09-21 01:18:42.733000 [ 590 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:18:42.733000 [ 590 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.764000 [ 590 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:18:42.765000 [ 590 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) _____________ ERROR at setup of test_embedded_database_and_tables ______________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 3508 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed ________________ ERROR at setup of test_multiple_schemes_tables ________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module") def started_cluster(): try: cluster = ClickHouseCluster(__file__) cluster.add_instance( "node1", main_configs=[], user_configs=[], image="clickhouse/integration-test-with-unity-catalog", with_installed_binary=False, tag=os.environ.get("DOCKER_BASE_WITH_UNITY_CATALOG_TAG", "latest") ) logging.info("Starting cluster...") > cluster.start() test_database_delta/test.py:43: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 3508 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed __________________ ERROR at setup of test_different_versions ___________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 @pytest.fixture(scope="module", autouse=True) def start_cluster(): try: > cluster.start() test_backup_restore_on_cluster/test_different_versions.py:44: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:2779: in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) helpers/retry_decorator.py:20: in inner func(*args, **kwargs) helpers/cluster.py:132: in run_and_check res = subprocess.run( /usr/lib/python3.10/subprocess.py:505: in run stdout, stderr = process.communicate(input, timeout=timeout) /usr/lib/python3.10/subprocess.py:1154: in communicate stdout, stderr = self._communicate(input, endtime, timeout) /usr/lib/python3.10/subprocess.py:2021: in _communicate ready = selector.select(timeout) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = , timeout = 192947 def select(self, timeout=None): # This is shared between poll() and epoll(). # epoll() has a different signature and handling of timeout parameter. if timeout is None: timeout = None elif timeout <= 0: timeout = 0 else: # poll() has a resolution of 1 millisecond, round away from # zero to wait *at least* timeout seconds. timeout = math.ceil(timeout * 1e3) ready = [] try: > fd_event_list = self._selector.poll(timeout) E Failed: Timeout >900.0s /usr/lib/python3.10/selectors.py:416: Failed ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ~~~~~~~~~~~~~~~~~~~~~ Stack of (140474758366784) ~~~~~~~~~~~~~~~~~~~~~ File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 411, in _perform_spawn reply.run() File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 341, in run self._result = func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 1160, in _thread_receiver msg = Message.from_io(io) File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 567, in from_io header = io.read(9) # type 1, channel 4, payload 4 File "/usr/local/lib/python3.10/dist-packages/execnet/gateway_base.py", line 534, in read data = self._read(numbytes - len(buf)) ------------------------------ Captured log setup ------------------------------ 2026-09-21 01:03:42.372000 [ 593 ] DEBUG : Command:[docker ps | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.405000 [ 593 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.405000 [ 593 ] DEBUG : No running containers (conftest.py:95, cleanup_environment) 2026-09-21 01:03:42.405000 [ 593 ] DEBUG : Pruning Docker networks (conftest.py:97, cleanup_environment) 2026-09-21 01:03:42.405000 [ 593 ] DEBUG : Command:[docker network prune --force] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.436000 [ 593 ] DEBUG : Command:[sysctl net.ipv4.ip_local_port_range='55000 65535'] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.439000 [ 593 ] DEBUG : Stdout:net.ipv4.ip_local_port_range = 55000 65535 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.440000 [ 593 ] INFO : Running tests in /ClickHouse/tests/integration/test_backup_restore_on_cluster/test_different_versions.py (cluster.py:2738, start) 2026-09-21 01:03:42.440000 [ 593 ] DEBUG : Cluster start called. is_up=False (cluster.py:2745, start) 2026-09-21 01:03:42.468000 [ 593 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:03:42.498000 [ 593 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:03:42.528000 [ 593 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:03:42.529000 [ 593 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:03:42.557000 [ 593 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:03:42.585000 [ 593 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:03:42.610000 [ 593 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:03:42.610000 [ 593 ] DEBUG : Command:[docker container list --all --filter name='^/roottestbackuprestoreonclusterdifferentversions-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.638000 [ 593 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:03:42.638000 [ 593 ] DEBUG : No running containers for project: roottestbackuprestoreonclusterdifferentversions-gw1 (cluster.py:879, cleanup) 2026-09-21 01:03:42.638000 [ 593 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:03:42.669000 [ 593 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:03:42.669000 [ 593 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.700000 [ 593 ] DEBUG : Stderr:Error response from daemon: a prune operation is already running (cluster.py:147, run_and_check) 2026-09-21 01:03:42.701000 [ 593 ] DEBUG : Exitcode:1 (cluster.py:149, run_and_check) 2026-09-21 01:03:42.701000 [ 593 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:03:42.701000 [ 593 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:03:42.727000 [ 593 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:03:42.727000 [ 593 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-09-21 01:03:42.727000 [ 593 ] DEBUG : Setup directory for instance: new_node (cluster.py:2758, start) 2026-09-21 01:03:42.729000 [ 593 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 01:03:42.729000 [ 593 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 01:03:42.730000 [ 593 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 01:03:42.731000 [ 593 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 01:03:42.732000 [ 593 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster_different_versions.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 01:03:42.732000 [ 593 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/database (cluster.py:4758, create_dir) 2026-09-21 01:03:42.732000 [ 593 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/logs (cluster.py:4769, create_dir) 2026-09-21 01:03:42.733000 [ 593 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-09-21 01:03:42.733000 [ 593 ] INFO : external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/backups (cluster.py:4879, create_dir) 2026-09-21 01:03:42.733000 [ 593 ] DEBUG : Setup directory for instance: old_node (cluster.py:2758, start) 2026-09-21 01:03:42.734000 [ 593 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4628, create_dir) 2026-09-21 01:03:42.734000 [ 593 ] DEBUG : Create directory for common tests configuration (cluster.py:4633, create_dir) 2026-09-21 01:03:42.735000 [ 593 ] DEBUG : Copy common configuration from helpers (cluster.py:4653, create_dir) 2026-09-21 01:03:42.735000 [ 593 ] DEBUG : Generate and write macros file (cluster.py:4705, create_dir) 2026-09-21 01:03:42.736000 [ 593 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/backups_disk.xml', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/configs/cluster_different_versions.xml'] to /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/configs/config.d (cluster.py:4741, create_dir) 2026-09-21 01:03:42.736000 [ 593 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/database (cluster.py:4758, create_dir) 2026-09-21 01:03:42.736000 [ 593 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/logs (cluster.py:4769, create_dir) 2026-09-21 01:03:42.737000 [ 593 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log", "--"] (cluster.py:4850, create_dir) 2026-09-21 01:03:42.737000 [ 593 ] INFO : external_dir_abs_path=/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/backups (cluster.py:4879, create_dir) 2026-09-21 01:03:42.737000 [ 593 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw', 'keeper_binary': '/clickhouse', 'keeper_cmd_prefix': 'clickhouse keeper', 'image': 'altinityinfra/integration-test:5ccda723c1fc', 'user': '0', 'keeper_fs': 'bind', 'keeper_logs_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper1/log', 'keeper_config_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper1/config', 'keeper_db_dir1': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper1/coordination', 'keeper_logs_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper2/log', 'keeper_config_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper2/config', 'keeper_db_dir2': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper2/coordination', 'keeper_logs_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper3/log', 'keeper_config_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper3/config', 'keeper_db_dir3': '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/keeper3/coordination'} stored in /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env (cluster.py:96, _create_env_file) 2026-09-21 01:03:42.738000 [ 593 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 01:03:42.738000 [ 593 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 01:03:42.738000 [ 593 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-09-21 01:03:42.738000 [ 593 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-09-21 01:03:42.754000 [ 593 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 826 (connectionpool.py:547, _make_request) 2026-09-21 01:03:42.754000 [ 593 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw1 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:08:42.858000 [ 593 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env', '--project-name', 'roottestbackuprestoreonclusterdifferentversions-gw1', '--file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-09-21 01:08:45.358000 [ 593 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw1 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:13:45.383000 [ 593 ] INFO : Got exception pulling images: Command '['docker', 'compose', '--env-file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env', '--project-name', 'roottestbackuprestoreonclusterdifferentversions-gw1', '--file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/docker-compose.yml', '--file', '/ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml', '--file', '/ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/docker-compose.yml', 'pull']' timed out after 300 seconds (cluster.py:2775, logging_pulling_images) 2026-09-21 01:13:48.744000 [ 593 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/.env --project-name roottestbackuprestoreonclusterdifferentversions-gw1 --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/new_node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/test_backup_restore_on_cluster/_instances-different_versions-1-gw1/old_node/docker-compose.yml pull] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.374000 [ 593 ] DEBUG : Failed to start cluster: (cluster.py:3169, start) 2026-09-21 01:18:42.374000 [ 593 ] DEBUG : Timeout >900.0s (cluster.py:3170, start) 2026-09-21 01:18:42.376000 [ 593 ] DEBUG : Traceback (most recent call last): File "/ClickHouse/tests/integration/helpers/cluster.py", line 2779, in start retry(log_function=logging_pulling_images)(run_and_check, images_pull_cmd) File "/ClickHouse/tests/integration/helpers/retry_decorator.py", line 20, in inner func(*args, **kwargs) File "/ClickHouse/tests/integration/helpers/cluster.py", line 132, in run_and_check res = subprocess.run( File "/usr/lib/python3.10/subprocess.py", line 505, in run stdout, stderr = process.communicate(input, timeout=timeout) File "/usr/lib/python3.10/subprocess.py", line 1154, in communicate stdout, stderr = self._communicate(input, endtime, timeout) File "/usr/lib/python3.10/subprocess.py", line 2021, in _communicate ready = selector.select(timeout) File "/usr/lib/python3.10/selectors.py", line 416, in select fd_event_list = self._selector.poll(timeout) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 313, in handler timeout_sigalrm(item, settings) File "/usr/local/lib/python3.10/dist-packages/pytest_timeout.py", line 498, in timeout_sigalrm pytest.fail("Timeout >%ss" % settings.timeout) File "/usr/local/lib/python3.10/dist-packages/_pytest/outcomes.py", line 198, in fail raise Failed(msg=reason, pytrace=pytrace) Failed: Timeout >900.0s (cluster.py:3171, start) 2026-09-21 01:18:42.377000 [ 593 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-09-21 01:18:42.377000 [ 593 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:18:42.403000 [ 593 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:18:42.430000 [ 593 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:18:42.451000 [ 593 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:18:42.451000 [ 593 ] DEBUG : Command:[docker container list --all --filter name='^/roottestbackuprestoreonclusterdifferentversions-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:18:42.473000 [ 593 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:18:42.473000 [ 593 ] DEBUG : No running containers for project: roottestbackuprestoreonclusterdifferentversions-gw1 (cluster.py:879, cleanup) 2026-09-21 01:18:42.473000 [ 593 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:18:42.503000 [ 593 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:18:42.503000 [ 593 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:18:44.414000 [ 593 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 01:18:44.414000 [ 593 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 01:18:44.414000 [ 593 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:18:44.414000 [ 593 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:18:44.447000 [ 593 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:18:44.447000 [ 593 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) 2026-09-21 01:18:44.447000 [ 593 ] WARNING : docker compose up was not called. Trying to export docker.log for running containers (cluster.py:3252, shutdown) 2026-09-21 01:18:44.448000 [ 593 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2026-09-21 01:18:44.480000 [ 593 ] DEBUG : Docker networks for project roottestbackuprestoreonclusterdifferentversions-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2026-09-21 01:18:44.511000 [ 593 ] DEBUG : Docker containers for project roottestbackuprestoreonclusterdifferentversions-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2026-09-21 01:18:44.538000 [ 593 ] DEBUG : Docker volumes for project roottestbackuprestoreonclusterdifferentversions-gw1 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2026-09-21 01:18:44.539000 [ 593 ] DEBUG : Command:[docker container list --all --filter name='^/roottestbackuprestoreonclusterdifferentversions-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2026-09-21 01:18:44.573000 [ 593 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2026-09-21 01:18:44.573000 [ 593 ] DEBUG : No running containers for project: roottestbackuprestoreonclusterdifferentversions-gw1 (cluster.py:879, cleanup) 2026-09-21 01:18:44.573000 [ 593 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2026-09-21 01:18:44.606000 [ 593 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2026-09-21 01:18:44.607000 [ 593 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2026-09-21 01:18:44.644000 [ 593 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2026-09-21 01:18:44.645000 [ 593 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2026-09-21 01:18:44.645000 [ 593 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2026-09-21 01:18:44.645000 [ 593 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2026-09-21 01:18:44.676000 [ 593 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2026-09-21 01:18:44.676000 [ 593 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) ----------------- generated report log file: parallel0_1.jsonl ----------------- ============================== slowest durations =============================== 902.31s setup test_backup_restore_on_cluster/test_different_versions.py::test_different_versions 900.39s setup test_database_delta/test.py::test_complex_table_schema 14.68s setup test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 4.45s teardown test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 0.32s call test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active 0.00s teardown test_database_delta/test.py::test_complex_table_schema 0.00s teardown test_database_delta/test.py::test_multiple_schemes_tables 0.00s teardown test_backup_restore_on_cluster/test_different_versions.py::test_different_versions 0.00s setup test_database_delta/test.py::test_embedded_database_and_tables 0.00s teardown test_database_delta/test.py::test_embedded_database_and_tables 0.00s setup test_database_delta/test.py::test_multiple_schemes_tables =========================== short test summary info ============================ ERROR test_database_delta/test.py::test_complex_table_schema - Failed: Timeou... ERROR test_database_delta/test.py::test_embedded_database_and_tables - Failed... ERROR test_database_delta/test.py::test_multiple_schemes_tables - Failed: Tim... ERROR test_backup_restore_on_cluster/test_different_versions.py::test_different_versions SKIPPED [1] test_asynchronous_metric_jemalloc_profile_active/test.py:30: Disabled for sanitizers =================== 1 skipped, 4 errors in 904.37s (0:15:04) =================== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_39dq2n --privileged --dns-search='.' --memory=30703239168 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel0_1.jsonl --report-log-exclude-logs-on-passed-tests test_asynchronous_metric_jemalloc_profile_active/test.py::test_asynchronous_metric_jemalloc_profile_active test_backup_restore_on_cluster/test_different_versions.py::test_different_versions test_database_delta/test.py::test_complex_table_schema test_database_delta/test.py::test_embedded_database_and_tables test_database_delta/test.py::test_multiple_schemes_tables -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.